Multi-module device including a printer module

ABSTRACT

A multi-module device having first and second modules is disclosed. The first and second modules communicate through a serial bus. The first module has a memory for storing image data in a RGB format and a processing unit for converting the image data from the RGB format to a L*a*b* format, before outputting image data in the L*a*b* format through the serial bus. The second module receives the image data in the L*a*b* format, converts the image data to a CMY format and prints the image data in the CMY format.

CROSS REFERENCES TO RELATED APPLICATION

The present application is a continuation of U.S. application Ser. No.12/106,331 filed on Apr. 21, 2008, which is a continuation of U.S.application Ser. No. 11/442,132 filed on May 30, 2006, now issued asU.S. Pat. No. 7,372,601, which is a continuation of U.S. applicationSer. No. 10/636,262 filed on Aug. 8, 2003, now issued as U.S. Pat. No.7,061,650, which is a divisional of U.S. application Ser. No. 09/575,122filed on May 23, 2000, now issued as U.S. Pat. No. 6,975,429, the entirecontents of which are herein incorporated by reference.

FIELD OF THE INVENTION

The invention relates to a compact printer system able to printfull-color, business card size documents from a device about the size ofa pen. The system includes various hot-connectable modules that providea range of functions. In particular the invention relates to a methodof, and an apparatus for, image conversion.

Reference may be had to co-pending applications claiming priority fromAustralian Provisional Patent Application number PQ0560 dated 25 May1999. The co-pending applications describe related modules and methodsfor implementing the compact printer system. The co-pending applicationsare as follows:

USSN Our Title 6,924,907 Compact Color Printer Module 6,712,452 ModularCompact Printer System 6,416,160 Nozzle Capping Mechanism 6,238,043 InkCartridge for Compact Printer System 6,958,826 Controller for PrinterModule 6,812,972 Camera Module for Compact Printer System 6,553,459Memory Module for Compact Printer System 6,967,741 Effects Module forCompact Printer System 6,956,669 Effects Processor for Effects Module6,903,766 Timer Module for Compact Printer System 6,804,026 ColorConversion Method for Compact Printer System 7,259,889 Method andApparatus of Dithering

BACKGROUND OF THE INVENTION

Microelectronic manufacturing techniques have led to the miniaturizationof numerous devices. Mobile phones, personal digital assistant devices,and digital cameras are very common examples of the miniaturizationtrend.

One device that has not seen the advantage of microelectronicmanufacturing techniques is the printer. Commercially available printersare large compared to many of the devices they could support. Forinstance, it is impractical to carry a color printer for the purpose ofinstantly printing photographs taken with known compact digital cameras.

A compact printhead has been described in co-pending United Statespatent applications filed simultaneously to the present application andhereby incorporated by cross reference:

USSN Title 7,018,016 Fluidic seal for an ink jet nozzle assembly6,428,133 Ink jet printhead having a moving nozzle with an externallyarranged actuator 6,526,658 Method of manufacture of an ink jetprinthead having a moving nozzle with an externally arranged actuator6,328,417 Ink jet printhead nozzle array 6,390,591 Nozzle guard for anink jet printhead

In order to put a compact printer system into practical effect it isnecessary to be able to capture an image for printing. An image can becaptured by various forms of sensors but a popular sensor is a Bayercolor filter array (CFA). The captured image must be processed to asuitable form for printing. Various techniques for processing the BayerCFA image are known but they suffer from various drawbacks that make analternative method desirable.

SUMMARY OF THE INVENTION

According to an aspect of the present invention there is provided amulti-module device comprising:

a first module having a memory for storing image data in a RGB format, aprocessing unit for converting the image data from the RGB format to aL*a*b* format, and a serial bus for outputting image data in the L*a*b*format; and

a second module in communication with the serial bus for receiving theimage data in the L*a*b* format, for converting the image data to a CMYformat and for printing the image data in the CMY format.

Further features of the invention will be evident from the followingdescription.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to assist with describing preferred embodiments of theinvention, reference will be made to the following figures in which:

FIG. 1 is a printer module;

FIG. 2 is a camera module;

FIG. 3 is a memory module;

FIG. 4 is a communication module;

FIG. 5 is a flash module;

FIG. 6 is a timer module;

FIG. 7 is a laser module;

FIG. 8 is an effects module;

FIG. 9 is a characters module;

FIG. 10 is an adaptor module;

FIG. 11 is a pen module;

FIG. 12 is a dispenser module;

FIG. 13 is a first compact printer configuration;

FIG. 14 is a second compact printer configuration;

FIG. 15 is a third compact printer configuration;

FIG. 16 is a fourth compact printer configuration;

FIG. 17 is a 2G mosaic pattern in a Bayer color filter array (CFA);

FIG. 18 shows color plane pixels in relation to full color filter arrayresolution;

FIG. 19 illustrates an image reconstruction and resampling method forthe green plane;

FIG. 20 shows the calculation of the distance between rotated pixels inthe green plane;

FIG. 21 shows mapping movement in unrotated CFA space to rotated CFAspace.

FIG. 22 shows the steps in the resample process; and

FIG. 23 is a block schematic diagram of a convolve unit for theresampling process.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to FIGS. 1 to 12, there are shown various modules thattogether form a compact printer system. Individual modules can beattached and detached from the compact printer configuration to allow auser-definable solution to business-card sized printing. Images can alsobe transferred from one compact printer to another without the use of asecondary computer system. Modules have a minimal user-interface toallow straightforward interaction.

A compact printer system configuration consists of a number of compactprinter modules connected together. Each compact printer module has afunction that contributes to the overall functionality of the particularcompact printer configuration. Each compact printer module is typicallyshaped like part of a pen, physically connecting with other compactprinter modules to form the complete pen-shaped device. The length ofthe compact printer device depends on the number and type of compactprinter modules connected. The functionality of a compact printerconfiguration depends on the compact printer modules in the givenconfiguration.

The compact printer modules connect both physically and logically. Thephysical connection allows modules to be connected in any order, and thelogical connection is taken care of by the compact printer Serial Bus—abus that provides power, allows the modules to self configure andprovides for the transfer of data.

In terms of physical connection, most compact printer modules consist ofa central body, a male connector at one end, and a female connector atthe other. Since most modules have both a male and female connector, themodules can typically be connected in any order. Certain modules onlyhave a male or a female connector, but this is determined by thefunction of the module. Adaptor modules allow these single-connectormodules to be connected at either end of a given compact printerconfiguration.

A four wire physical connection between all the compact printer modulesprovides the logical connection between them in the form of the compactprinter Serial Bus. The compact printer Serial Bus provides power toeach module, and provides the means by which data is transferred betweenmodules. Importantly, the compact printer Serial Bus and accompanyingprotocol provides the means by which the compact printer systemauto-configures, reducing the user-interface burden on the end-user.

Compact printer modules can be grouped into three types:

-   -   image processing modules including a Printer Module (FIG. 1), a        Camera Module (FIG. 2), and a Memory Module (FIG. 3). Image        processing modules are primarily what sets the compact printer        system apart from other pen-like devices. Image processing        modules capture, print, store or manipulate photographic images;    -   housekeeping modules including an Adapter Module (FIG. 10), an        Effects Module (FIG. 8), a Communications Module (FIG. 4), and a        Timer Module (FIG. 6). Housekeeping modules provide services to        other modules or extended functionality to other modules; and    -   isolated modules including a Pen Module (FIG. 11) and a Laser        Module (FIG. 7). Isolated modules are those that attach to the        compact printer system but are completely independent of any        other module. They do not necessarily require power, and may        even provide their own power. Isolated Modules are defined        because the functionality they provide is typically incorporated        into other pen-like devices.

Although housekeeping modules and isolated modules are useful componentsin a compact printer system, they are extras in a system dedicated toimage processing and photographic manipulation. Life size (1:1)illustrations of the compact printer modules are shown in FIGS. 1 to 12,and example configurations produced by connecting various modulestogether are shown in FIGS. 13 to 16.

FIG. 1 shows a printer module that incorporates a compact printheaddescribed in co-pending United States patent applications listed in theBackground section of this application, incorporated herewith byreference, and referred to herewith as a Memjet printhead. The Memjetprinthead is a drop-on-demand 1600 dpi inkjet printer that producesbi-level dots in up to 4 colors to produce a printed page of aparticular width. Since the printhead prints dots at 1600 dpi, each dotis approximately 22.5 μm in diameter, and spaced 15.875 μm apart.Because the printing is bi-level, the input image should be dithered orerror-diffused for best results. Typically a Memjet printhead for aparticular application is page-width. This enables the printhead to bestationary and allows the paper to move past the printhead. A Memjetprinthead is composed of a number of identical ½ inch Memjet segments.

The printer module 10 comprises a body 11 housing the Memjet printhead.Power is supplied by a three volt battery housed in battery compartment12. The printhead is activated to commence printing when a business card(or similar sized printable media) is inserted into slot 13. Maleconnector 14 and female connector 15 facilitate connection of othermodules to the printer module 10.

FIG. 2 shows a camera module 20. The camera module provides apoint-and-shoot camera component to the compact printer system as ameans of capturing images. The camera module comprises a body 21 havinga female connector 22. A lens 23 directs an image to an image sensor andspecialized image processing chip within the camera 24. A conventionalview finder 25 is provided as well as a lens cap 26. An image iscaptured when the Take button 27 is pushed. Captured images aretransferred to the Printer Module 10 for subsequent printing,manipulation, or storage. The Camera Module also contains a self-timermode similar to that found on regular cameras.

FIG. 3 shows a Memory Module 30 comprising a body 31, LCD 32, IN button33, OUT button 34 and SELECT button 35. The Memory Module 30 is astandard module used for storing photographic images captured by theCamera 20. The memory module stores 48 images, each of which can beaccessed either at full resolution or at thumbnail resolution. Fullresolution provides read and write access to individual images, andthumbnail resolution provides read access to 16 images at once inthumbnail form.

The Memory Module 30 attaches to other modules via a female connector 36or male connector 37. The male and female connectors allow the module tobe connected at either end of a configuration. Power is provided fromthe Printer Module 10 via the Serial Bus.

A Communications Module 40 is shown in FIG. 4. The communications module40 consists of a connector 41 and a cable 42 that terminates in anappropriate connector for a computer port, such as a USB port, RS232serial port or parallel port. The Communications Module 40 allows thecompact printer system to be connected to a computer. When so connected,images can be transferred between the computer and the various modulesof the compact printer system. The communications module allows capturedimages to be downloaded to the computer, and new images for printing tobe uploaded into the printer module 10.

A Flash Module 50 is shown in FIG. 5. The Flash Module 50 is used togenerate a flash with flash cell 51 when taking photographs with theCamera Module 20. The Flash Module attaches to other modules via femaleconnector 52 and male connector 53. It contains its own power source.The Flash Module is automatically selected by the Camera Module whenrequired. A simple switch allows the Flash Module to be explicitlyturned off to maximize battery life.

FIG. 6 shows a Timer Module 60 that is used to automate the taking ofmultiple photos with the Camera Module 20, each photo separated by aspecific time interval. The captured photos are stored in Memory Module30. Any flash requirements are handled by the Camera Module 20, and cantherefore be ignored by the Timer Module. The Timer Module 60 consistsof a body 61 housing a LCD 62, START/STOP button 63 and UNITS button 64.A SELECT button 65 allows the user to select time units and the numberof units are set by UNITS button 64. The Timer Module 60 includes a maleconnector 66 and female connector 67. The Timer Module takes its powerfrom the Printer Module 10 via the Serial Bus.

A Laser Module 70 is shown in FIG. 7. The Laser Module 70 consists of abody 71 containing a conventional laser pointer operated by button 72.As the Laser Module is a terminal module it only has one connector,which in the example is a male connector 73. The Laser Module is anisolated module, in that it does not perform any image capture, storage,or processing. It exists as a functional addition to the compact printersystem. It is provided because laser pointer services are typicallyincorporated into other pen-like devices. The Laser Module contains itsown power supply and does not appear as a device on the Serial Bus.

The Effects Module shown in FIG. 8 is an image processing module. Itallows a user to select a number of effects and applies them to thecurrent image stored in the Printer Module 10. The effects includeborders, clip-art, captions, warps, color changes, and painting styles.The Effects Module comprises a body 81 housing custom electronics and aLCD 82. A CHOOSE button 83 allows a user to choose between a number ofdifferent types of effects. A SELECT button 84 allows the user to selectone effect from the number of effects of the chosen type. Pressing theAPPLY button 85 applies the effect to image stored in the Printer Module10. The Effects Module obtains power from the Serial Bus. Male connector86 and female connector 87 allow the Effects Module to be connected toother compact printer system modules.

FIG. 9 shows a Character Module 90 that is a special type of EffectsModule (described above) that only contains character clip-art effectsof a given topic or genre. Examples include The Simpsons®, Star Wars®,Batman®, and Dilbert® as well as company specific modules for McDonalds®etc. As such it is an image processing module. It consists of a body 91housing custom electronics and a LCD 92. SELECT button 93 allows theuser to choose the effect that is to be applied with APPLY button 94.The Character Module obtains power from the Serial Bus through maleconnector 95 and female connector 96.

The Adaptor Module 100, shown in FIG. 10, is a female/female connectorthat allows connection between two modules that terminate in maleconnectors. A male/male connector (not shown) allows connection betweentwo modules that terminate in female connectors. The Adaptor Module is ahousekeeping module, in that it facilitates the use of other modules,and does not perform any specific processing of its own.

All “through” modules have a male connector at one end, and a femaleconnector at the other end. The modules can therefore be chainedtogether, with each module connected at either end of the chain. Howeversome modules, such as the Laser Module 70, are terminating modules, andtherefore have either a male or female connector only. Suchsingle-connector modules can only be connected at one end of the chain.If two such modules are to be connected at the one time, an AdaptorModule 100 is required.

FIG. 11 shows a Pen Module 110 which is a pen in a module form. It is anisolated module in that it attaches to the compact printer system but iscompletely independent of any other module. It does not consume orrequire any power. The Pen Module is defined because it is a convenientextension of a pen shaped, pen sized device. It may also come with a cap111. The cap may be used to keep terminating connectors clean in thecase where the chain ends with a connector rather than a terminatingmodule.

To assist with accurately feeding a business card sized print media intoslot 13 of the printer module 10, a dispenser module 120 is provided asshown in FIG. 12. The dispenser module 120 comprises a body 121 thatholds a store of business card sized print media. A Printer Module 10locates into socket 122 on the dispenser module 120. When correctlyaligned, a card dispensed from the dispenser module by slider 123 entersslot 13 and is printed.

In the sense that a minimum configuration compact printer system must beable to print out photos, a minimum compact printer configurationcontains at least a Printer Module 10. The Printer Module holds a singlephotographic image that can be printed out via its Memjet printer. Italso contains the 3V battery required to power the compact printersystem.

In this minimum configuration, the user is only able to print outphotos. Each time a user inserts a business card 130 into the slot inthe Printer Module, the image in the Printer Module is printed onto thecard. The same image is printed each time a business card is insertedinto the printer. In this minimum configuration there is no way for auser to change the image that is printed. The dispenser module 120 canbe used to feed cards 130 into the Printer Module with a minimum offuss, as shown in FIG. 13.

By connecting a Camera Module 20 to the minimum configuration compactprinter system the user now has an instant printing digital camera in apen, as shown in FIG. 14. The Camera Module 20 provides the mechanismfor capturing images and the Printer Module 10 provides the mechanismfor printing them out. The battery in the Printer Module provides powerfor both the camera and the printer.

When the user presses the “Take” button 27 on the Camera Module 20, theimage is captured by the camera 24 and transferred to the Printer Module10. Each time a business card is inserted into the printer the capturedimage is printed out. If the user presses “Take” on the Camera Moduleagain, the old image in the Printer Module is replaced by the new image.

If the Camera Module is subsequently detached from the compact printersystem, the captured image remains in the Printer Module, and can beprinted out as many times as desired. The Camera Module is simply thereto capture images to be placed in the Printer Module.

FIG. 15 shows a further configuration in which a Memory Module 30 isconnected to the configuration of FIG. 14. In the embodiment of FIG. 15,the user has the ability to transfer images between the Printer Module10 and a storage area contained in the Memory Module 30. The userselects the image number on the Memory Module, and then either sendsthat image to the Printer Module (replacing whatever image was alreadystored there), or brings the current image from the Printer Module tothe specified image number in the Memory Module. The Memory Module alsoprovides a way of sending sets of thumbnail images to the PrinterModule.

Multiple Memory Modules can be included in a given system, extending thenumber of images that can be stored. A given Memory Module can bedisconnected from one compact printer system and connected to anotherfor subsequent image printing.

With the Camera Module 20 attached to a Memory Module/Printer Modulecompact printer system, as shown in FIG. 15, the user can “Take” animage with the Camera Module, then transfer it to the specified imagenumber in the Memory Module. The captured images can then be printed outin any order.

By connecting a Communications Module 40 to the minimum configurationcompact printer system, the user gains the ability to transfer imagesbetween a PC and the compact printer system. FIG. 16 shows theconfiguration of FIG. 15 with the addition of a Communications Module40. The Communications Module makes the Printer Module 10 and any MemoryModules 30 visible to an external computer system. This allows thedownload or uploading of images. The communications module also allowscomputer control of any connected compact printer modules, such as theCamera Module 20.

In the general case, the Printer Module holds the “current” image, andthe other modules function with respect to this central repository ofthe current image. The Printer Module is therefore the central locationfor image interchange in the compact printer system, and the PrinterModule provides a service to other modules as specified by userinteraction.

A given module may act as an image source. It therefore has the abilityto transfer an image to the Printer Module. A different module may actas an image store. It therefore has the ability to read the image fromthe Printer Module. Some modules act as both image store and imagesource. These modules can both read images from and write images to thePrinter Module's current image.

The standard image type has a single conceptual definition. The imagedefinition is derived from the physical attributes of the printhead usedin the Printer Module. The printhead is 2 inches wide and prints at 1600dpi in cyan, magenta and yellow bi-level dots. Consequently a printedimage from the compact printer system is 3200 bi-level dots wide.

The compact printer system prints on business card sized pages (85 mm×55mm). Since the printhead is 2 inches wide, the business cards areprinted such that 1 line of dots is 2 inches. 2 inches is 50.8 mm,leaving a 2 mm edge on a standard business-card sized page. The lengthof the image is derived from the same card size with a 2 mm edge.Consequently the printed image length is 81 mm, which equals 5100 1600dpi dots. The printed area of a page is therefore 81 mm×51 mm, or5100×3200 dots.

To obtain an integral contone to bi-level ratio a contone resolution of267 ppi (pixels per inch) is chosen. This yields a contone CMY page sizeof 850×534, and a contone to bi-level ratio of 1:6 in each dimension.This ratio of 1:6 provides no perceived loss of quality since the outputimage is bi-level.

The printhead prints dots in cyan, magenta, and yellow ink. The finaloutput to the printed page must therefore be in the gamut of theprinthead and take the attributes of the inks into account. It would atfirst seem reasonable to use the CMY color space to represent images.However, the printer's CMY color space does not have a linear response.This is definitely true of pigmented inks, and partially true fordye-based inks. The individual color profile of a particular device(input and output) can vary considerably. Image capture devices (such asdigital cameras) typically work in RGB (red green blue) color space, andeach sensor will have its own color response characteristics.

Consequently, to allow for accurate conversion, as well as to allow forfuture image sensors, inks, and printers, the CIE L*a*b* color model[CIE, 1986, CIE 15.2 Colorimetry: Technical Report (2^(nd) Edition),Commission Internationale De l'Eclairage] is used for the compactprinter system. L*a*b* is well defined, perceptually linear, and is asuperset of other traditional color spaces (such as CMY, RGB, and HSV).

The Printer Module must therefore be capable of converting L*a*b* imagesto the particular peculiarities of its CMY color space. However, sincethe compact printer system allows for connectivity to PCs, it is quitereasonable to also allow highly accurate color matching between screenand printer to be performed on the PC. However the printer driver or PCprogram must output L*a*b*.

Each pixel of a compact printer image is therefore represented by 24bits: 8 bits each of L*, a*, and b*. The total image size is therefore1,361,700 bytes (850×534×3).

Each image processing module is able to access the image stored in thePrinter Module. The access is either to read the image from the PrinterModule, or to write a new image to the Printer Module.

The communications protocol for image access to the Printer Moduleprovides a choice of internal image organization. Images can be accessedeither as 850×534 or as 534×850. They can also be accessed ininterleaved or planar format. When accessed as interleaved, each pixelin the image is read or written as 24 bits: 8 bits each of L*, a*, b*.When accessed as planar, each of the color planes can be read or writtenindependently. The entire image of L* pixels, a* pixels or b* pixels canbe read or written at a time.

In a preferred embodiment, the Camera Module includes the imageprocessing chip which is responsible for enhancing the captured image toproduce a high quality L*a*b* image for transmission to the PrinterModule 10.

The input image comes from an image sensor. Although a variety of imagesensors are available, the Bayer color filter array (CFA) is preferred.The Bayer CFA has a number of attributes that are described here.

The image captured by the CMOS sensor (via a taking lens) is assumed tohave been sufficiently filtered so as to remove any aliasing artifacts.The sensor itself has an aspect ratio of approximately 3:2, with aresolution of 850×534 samples to match the image resolution of the finaloutput image. The most likely pixel arrangement is the Bayer colorfilter array (CFA), with each 2×2 pixel block arranged in a 2G mosaic asshown in FIG. 17. Other pixel arrangements are also possible and will besuitable for certain applications. For example, a 2R mosaic will enhancethe red end of the spectrum.

Each contone sample of R, G, or B (corresponding to red, green, and bluerespectively) is 10-bits. Note that each pixel of the mosaic containsinformation about only one of R, G, or B. Estimates of the missing colorinformation must be made before the image can be printed out.

Thus, the CFA only provides a single color component per pixel (x,y)coordinate. To produce the final image we need to have the other colorcomponent values at each pixel. With one color per pixel, we may havethe red component for a particular position, but we need to estimateblue and green. Or we may have green, and need to estimate red and blue.

Resampling is one of the steps in the image enhancement process. Todecide how best to resample, it is best to consider each color plane inrelation to the full CFA resolution (no rotation). This is shown in FIG.18.

Looking at the red and blue planes in FIG. 18, the full CFA resolutionversion of the color plane can be created by scaling up the number ofpixels sampled from the mosaic in each dimension by 2. The intermediatepixels can be generated by means of a reconstruction filter (such as aLanczos or Exponential filter). Only one dimension in the kernel isrequired, since the kernel is symmetric. Since red and blue havedifferent offsets in terms of their initial representation within theCFA sample space, the initial positions in the kernel will be different.

The following relationship holds for red and blue mapping of outputcoordinates (in 534 space) to input coordinates:x′=(x/ops)&y′=(y/ops)where

x, y=coordinate in output resolution space;

x′, y′=coordinate in input space;

ops=2=output resolution pixels per input space sample.

The number of output resolution pixels per sample, ops, is a constantvalue of 2. This means that given a starting position in input space, wecan generate a new line of CFA resolution pixels by adding a Δx and Δyof ½ and 0 respectively 533 times. The fractional part of x and y ininput space can be directly used for looking up the kernel coefficientsfor image reconstruction and resampling. Since Δx is ½, we only require2 sets of kernel coefficients.

Since the red and blue planes are scaled up, there will not be anyaliasing artifacts introduced by the resampling process.

The green plane cannot be simply scaled up in the same way as red orblue, since each line of the green plane represents differentpixels—either the odd or even pixels on alternate lines. Although interms of the number of pixels it is representative to say the greenimage is 425×534, the image could equally be said to be 850×267. Thisconfusion arises because of the checkerboard nature of the green pixels,where the distance between pixels is not equal in x and y dimensions,and does not map well to image reconstruction or resampling. The numberof interpolation methods used by other systems for green planereconstruction is testimony to this—from nearest neighbor replication tolinear interpolation to bi-linear interpolation and heuristicreconstruction.

The mapping of output coordinates (in 534 space) to input coordinates isconceptually the same for green as it is for red and blue. For the greenplane the following relationship holds:x′=(x/ops)&y′=(y/ops)where

x, y=coordinate in output resolution space;

x′, y′=coordinate in input space;

ops=1=output resolution pixels per input space sample.

-   -   Setting the number of output resolution pixels per sample, ops,        to 1 allows the direct usage of coordinates in CFA resolution        input space. However, once we have a coordinate in CFA        resolution input space, we cannot perform image reconstruction        and resampling on the samples in the same way as red or blue due        to the checkerboard nature of the green plane.

Instead, for the purposes of high quality image reconstruction andresampling, we can consider the green channel to be an image rotated by45 degrees. When we look at the pixels in this light, as shown in FIG.19, a high quality image reconstruction and resampling method becomesavailable.

Looking at FIG. 19, the distance between the sampled pixels in the X andY directions is now equal. The actual distance between sampled pixels is√2, as illustrated in FIG. 20.

Thus, the solution to resampling the green channel is to perform imagereconstruction and resampling in rotated space. Although the samereconstruction filter is used as for resampling red and blue, the kernelshould be different. This is because the relationship between thesampling rate for green and the highest frequency in the signal isdifferent to the relationship for the red and blue planes. In addition,the kernel should be normalized so that the √2 distance between samplesbecomes 1 as far as kernel coordinates go (the un-normalized distancesbetween resampling coordinates must still be used to determine whetheraliasing will occur however). Therefore we require two transformations:

-   -   The first is to map unrotated CFA space into rotated CFA space.        This can be accomplished by multiplying each ordinate by 1/√2,        since we are rotating by 45 degrees (cos 45°=sin 45°=1/√2).    -   The second is to scale the coordinates to match the normalized        kernel, which can be accomplished by multiplying each ordinate        by 1/√2.

These two transformations combine to create a multiplication factor of½. Consequently, as we advance in unrotated CFA space x by k, weincrease by k/2 in kernel x, and decrease by k/2 in kernel y. Similarly,as we advance in y by k, we increase by k/2 in kernel x and increase byk/2 in kernel y.

The relationships between these different coordinate systems can beillustrated by considering what occurs as we generate a line of outputpixels from a CFA space input image. Given a starting y ordinate in CFAinput space, we begin at x=0, and advance 850 times by 1, generating anew output pixel at each new location. The movement in unrotated CFAspace by 1 can be decomposed into a movement in x and a movement in y inrotated CFA space. The process is shown in FIG. 21.

Since cos 45°=sin 45°=1/√2, movement in unrotated CFA space by 1 equatesto equal movement in x and y by 1/√2 in rotated CFA space. This amountmust now be scaled to match the normalized kernel. The scaling equatesto another multiplication by 1/√2. Consequently, a movement of 1 inunrotated CFA space equates to a movement of ½ in kernel x and kernel y.

Since the Δ of ½ is less than 1, we are scaling up, and thereforealiasing will not occur for green resampling. In addition, a Δ of ½means that we only require 2 sets of kernel coefficients.

A reconstruction filter for the resampled red, blue and green channelsis required. The exact reconstruction filter to be used will depend on anumber of issues. There is always a trade off between the number ofsamples used in constructing the original signal, the time taken forsignal reconstruction, and the quality of the resampled image. Asatisfactory trade-off in this case is 5 pixel samples from thedimension being reconstructed, centered around the estimated position Xi.e. X−2, X−1, X, X+1, X+2. Due to the nature of reconstructing with 5sample points, only 4 coefficients for the entry in the convolutionkernel are required.

With generalized resampling, a kernel coefficient lookup table with nentries for each color component is created. Each entry has 4coefficients. For advancement in output space, the changes in outputspace are mapped to changes in input space and kernel space. The mostsignificant bits of the fractional component in the current kernel spaceare used to index into the kernel coefficients table. In this case, withred, green, and blue all requiring only 2 sets of kernel coefficientseach, only 2 entries in the table are required: an entry for 0, and anentry for ½.

The apparatus for implementing the aforementioned method of the presentinvention can be seen in FIG. 22. Address generation is not shown.

The resampling process can be seen as 3 sets of RGB generation. RGBvalues must effectively be produced for 3 pixel coordinates: the pixelin question, and the pixel above and below. The process of generating asingle RGB value can in turn be seen as 3 processes performed inparallel: the calculation of R, G, and B, all for a given pixelcoordinate. Due to the differences in resampling each channel, it isnecessary to effectively run three image reconstruction filters, one oneach channel of the image.

Consequently, calculation of the medium resolution R pixel is achievedby running an image reconstruction filter on the R data. Similarly,calculation of the medium resolution G and B pixels is achieved byrunning an image reconstruction filter on the G and B data. Although thekernels are symmetric in x and y, they are not the same for each colorplane. R and B are likely to be the same kernel due to their similarimage characteristics, but the G plane, due to the rotation required forimage reconstruction, must have a different kernel.

To calculate a given color plane's medium resolution pixel value, 24cycles are available (since 72 cycles in total are available forproducing the components of R, G and B). To apply the kernel to a 4×4sample area, the 1D kernel (indexed by x) is applied to each of the 4rows of 4 input samples. The 1D kernel (indexed by y) is then applied tothe resultant 4 pixel values. The final result is the output resampledpixel. Applying a single coefficient each cycle gives a total of 16cycles to generate the 4 intermediate values, and 4 cycles to generatethe final pixel value, for a total of 20 cycles.

The input pixels are each 10 bits (8:2), and kernel coefficients are 12bits. 14 bits of precision are kept during the 4 steps of eachapplication of the kernel (8:6), but only 10 bits are saved for theresult (8:2). Thus the same convolve engine can be used when convolvingin x and y. The final output of R, G, or B is only 8 bits. The convolveunit for the resampling process is shown in FIG. 23.

The process of resampling then, involves 20 cycles, as shown in Table 1.Note that the Row 1, Pixel 1 etc. refers to the input from Buffer 1,which is managed by a known addressing mechanism.

TABLE 1 The 20 Cycle Resample Cycle Kernel Apply Kernel to: Store Resultin 1 X[1] Row 1, Pixel 1 TMP 2 X[2] Row 1, Pixel 2 TMP 3 X[3] Row 1,Pixel 3 TMP 4 X[4] Row 1, Pixel 4 TMP, V1 5 X[1] Row 2, Pixel 1 TMP 6X[2] Row 2, Pixel 2 TMP 7 X[3] Row 2, Pixel 3 TMP 8 X[4] Row 2, Pixel 4TMP, V2 9 X[1] Row 3, Pixel 1 TMP 10 X[2] Row 3, Pixel 2 TMP 11 X[3] Row3, Pixel 3 TMP 12 X[4] Row 3, Pixel 4 TMP, V3 13 X[1] Row 4, Pixel 1 TMP14 X[2] Row 4, Pixel 2 TMP 15 X[3] Row 4, Pixel 3 TMP 16 X[4] Row 4,Pixel 4 TMP, V4 17 Y[1] V1 TMP 18 Y[2] V2 TMP 19 Y[3] V3 TMP 20 Y[4] V4TMP (for output)

Regarding the address generation for kernel lookup, because the scalingup is by 2, the delta values are ½. Consequently each kernel is 1dimensional, with 2 entries in the table. The most significant bit(truncated) of the fractional component in the current kernel space isused to index into the kernel coefficients table. For the first 16cycles, the X ordinate is used to index the kernel, while in the next 4cycles, the Y ordinate is used. Since the kernel is symmetric, the samekernel can be used for both X and Y.

For each of the 534 resampled values, 3 pixels need to be produced—thepixel in question, and the pixels above and below that pixel. Ratherthan generate a center pixel and then move up and down from that centerpixel, a pixel is generated and then two pixels below it are generated.The second pixel generated is taken to be the center pixel. Returning tothe original row, the next 3 pixels in the next output position are thengenerated. In this way, 3 pixels for each of the 534 positions aregenerated.

Address generation for the green sub-buffer within Buffer 1 is morecomplicated than the red and blue sub-buffers because, to resample thegreen channel, we must effectively rotate the channel by 45 degrees.Furthermore, there are twice as many green pixels than red or bluepixels. Resampling means the reading of more samples in the same amountof time—there are still 16 samples read to generate each pixel, butthere is a higher likelihood of advancing the buffer each time. Theexact likelihood depends on the scale factor used.

In the preferred embodiment, a RAM is used as a cyclical buffer for thegreen channel. The green sub-buffer is a 66 entry RAM with a logicalarrangement of 11 rows, each containing 6 entries.

The samples in Buffer 1 represent a checkerboard pattern in the CFA.Consequently, samples in one row (e.g. addresses 0, 11, 22, 33, 44, 55)may represent odd or even pixels, depending on the current line withinthe entire image. For example, for a basic 2×2 CFA cell, as shown inFIG. 17, with even rows having samples at odd pixels and odd rows havingsamples at even pixels, the converse would be true for a rotation of the2×2 CFA cell through 90 degrees.

Consequently, when a 4×4 sampling area is mapped onto the buffer, thereare two possibilities for the interpretation of the samples. As aresult, there are two types of addressing, depending on whether thecurrent line is represented by odd or even pixels. This means that evenrows with image rotation 0 will have the same addressing as odd rowswith image rotation 90 since they both hold odd pixels. Likewise, theodd rows with image rotation 0 will have the same addressing as evenrows with image rotation 90 since they both hold even pixels. This meansthe physical CFA orientation can be taken account of.

The actual 4×4 sampling window is the way the buffer is effectivelyrotated by 45 degrees, the 45 degree rotation being necessary foreffective resampling of the green channel.

Regarding the red and blue channels, the red and blue sub-buffers ofBuffer 1 are also RAMs accessed as cyclical buffers, as for the greensub-buffer. However, each buffer is 30 bytes, but has a logicalarrangement of 6 rows, each containing 6 entries.

The address generation mechanism for the red and blue channels isdifferent from the green channel. However, rather than design twoaddressing mechanisms, it is possible to apply the green addressingscheme to the red and blue channels, and simply use different values inlook up tables. This reduces design complexity.

For red and blue there is no scaling between movement in kernel spaceand movement in the input space. There is also no rotation.

The sampling process for a given 534 position line cannot begin untilthere are enough entries in Buffer 1, (placed there by aWhiteBalance/RangeExpansion Unit of an Image Enhancement Unit). Thiswill occur 128 cycles after the start of each new line. Obviously,green, red and blue generation must start at the same time.

Throughout the specification the aim has been to describe the preferredembodiments of the invention without limiting the invention to any oneembodiment or specific collection of features. Persons skilled in therelevant art may realize variations from the specific embodiments thatwill nonetheless fall within the scope of the invention.

1. A multi-module device comprising: a first module having an imagesensor for capturing image data in a Bayer color filter array (CFA)pixel arrangement, a memory for storing image data in a RGB format, aprocessing unit for converting the image data from the RGB format to aL*a*b* format, and a serial bus for outputting image data in the L*a*b*format; and a second module in communication with the serial bus forreceiving the image data in the L*a*b* format, for converting the imagedata to a CMY format and for printing the image data in the CMY format.2. The multi-module device of claim 1, wherein the image processing unitis further configured to transform the image data via white balance,range expansion, color conversion and image sharpening calculations. 3.The multi-module device of claim 2, wherein the color conversioncalculation by the image processing unit includes tri-linearinterpolation according to three 17×17×17×8-bit lookup tables for RGB toL*, RGB to a*, and RGB to b* respectively.
 4. The multi-module device ofclaim 2, wherein the image sharpening calculation by the imageprocessing unit includes a highpass filter using a 3×3 convolutionkernel for L*.
 5. The multi-module device of claim 1, wherein the firstand second modules interconnect through male or female connectors havinga four wire serial bus interface.
 6. The multi-module device of claim 1,wherein the image processing module is configured to resample red andblue planes of the captured image data by scaling up a number of pixelssampled from a mosaic of the captured RGB image in each plane dimensionby 2, with intermediate pixels generated by means of a reconstructionfilter.
 7. The multi-module device of claim 6, wherein the imageprocessing module is configured to resample a green plane of thecaptured image by mapping output coordinates in CMY space to inputcoordinates of the RGB image according to a particular relationship.